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FOREWORD 


In January of 1983, an Ad Hoc Student Evaluation Technical 
Advisory Committee was struck with the purpose of reviewing the technical 
aspects of the provincial Student Evaluation Program and making 
appropriate recommendations to the Director of Student Evaluation of 
Alberta Education. The terms of reference for the committee are 
presented in Appendix I. 


Five meetings were held in which the committee reviewed the 
procedures used by the Branch in the development, administration, and 
scoring of examinations. The committee also scrutinized the reporting of 
achievement test results significant at the provincial and local levels. 
At these meetings, Branch personnel knowledgeable in various aspects of 
the procedures described the work that had been done, with particular 
attention to the ways in which various testing elements had evolved. 


While the mandate of the committee extended to the investigation 
of all student evaluation programs, it became clear that only the 
achievement testing program was sufficiently well developed to warrant 
review. It is hoped, however, that some of the recommendations contained 
in this report will be useful for other programs. As a result of these 
deliberations, the committee's general view was that the Student 
Evaluation Branch was doing an excellent job of conducting the 
achievement testing program, within the time and personnel constraints 
that were imposed. Several areas, however, were identified as needing 
improvement. These areas are described in the remainder of the report. 
The recommendations were made within the constraints of the Ministerial 
mandate requiring that achievement test results be valid at the 
jurisdiction level. 


The report addresses eight distinct categories: the test 
development process, sampling, test security, test equating, standard 
setting, reporting of results, the achievement testing schedule, and 
operational studies in testing techniques. Recommendations are made 
within each category, and the rationale for each recommendation is 
provided. 


Finally, the report evolved from a consensus-seeking process: 
the committee members discussed, debated, and compromised on specific 
points, recognizing that the report served primarily an advisory 
purpose. Thus, the report does not wholly reflect the position of any 
one committee member or referent group. The committee hopes that this 
report will result in an increased understanding of the technical aspects 
of testing that will ultimately lead to better practices. 


J. Lawrence Tymko 
Committee Chairman 
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1.1 


1.2 


ISSUES AND RECOMMENDATIONS 


1. Test Development Process 


Issue 
Name of the Technical Review Committee 


The name "Technical Review Committee" does not reflect the 
function of the committee, which is to review the tests from all 
perspectives, not only from the technical point of view. This 
committee also examines various phases of the testing process in 
addition to the test itself. 


Recommendation 


That the name of the Technical Review Committee be changed to the 
Examination Review Committee. 


Issue 
Function of the Testing Review Committee 


Currently there is no requirement for a comprehensive review of 
the final form of a test. The Examination Review Committee should 
provide a final check on the quality of the test before it is 
administered. The review of each item and a check on its match 
with the blueprint category will ensure, insofar as is possible, 
high content validity. 


Recommendation 


That recommendation for formal approval of the final form be 
added to the functions of the Examination Review Committee as 
identified in the document “Examination Development Process” 
(Appendix III). This function involves reaching consensus on each 


of the following: 


individual items 

match of test to blueprint 
test organization and format 
reporting categories 
administration instructions 


Issue 
Teacher representation on the Examination Review Committee 


Teacher representation on the Examination Review Committee will 
help ensure that the achievement tests reflect program 
implementation. Although teachers construct the original items, 
these items often undergo significant revisions before appearing 
on the final form of the test. Therefore, it is important to have 
practising teachers on the committee reviewing the final form of 
the test. 


Recommendation 


That the number of teacher representatives nominated by the ATA 
be increased to three, at least one of whom is currently teaching 
the specific content covered by the test. 


Issue 
Technical advice for test development 


It is necessary to ensure high technical merit of instruments used 
in the achievement testing program. Procedures used in the 
development, validation, and administration of tests should be 
reviewed periodically, and revised to reflect current developments 
in psychometrics. The number of people in the Branch who have the 
technical skills required is small and their time is limited. The 
Branch procedures should be reviewed periodically by testing 
specialists external to the Branch. In addition to providing an 
external review, these experts can also augment the technical 
capability of the Branch when needed. 


Recommendation 


That a Technical Advisory Committee (TAC) of individuals 


knowledgeable in testing be established to conduct periodic 
reviews of testing procedures and to serve on Examination Review 
Committees. 


2, Sampling 


Issue 
Student sampling 


The mandate of the Branch requires that results be valid at the 
jurisdiction level. For larger jurisdictions this can be achieved 
by sampling. Sampling by class or by student could result in 
motivation problems in schools where some students are writing and 
some are not, and sample selection is complicated by difficulty in 
defining a class unit. Using the school as a sampling unit 
overcomes these problems and also simplifies the sampling and test 
administration procedures. 


Recommendation 


That each jurisdiction with a sufficient number of schools be 
allowed to choose to test all students or to test a sample 
selected on a school basis. 


Issue 
Curriculum sampling 


The SEB must provide information that can be used to make 
decisions affecting curriculum design and implementation, and 
resource allocation. This information must provide for a broad 
range of topics within each content area. Matrix sampling allows 
for more items to be used, thus covering a broader spectrum of the 
curriculum and increasing content validity. In addition, each 
specific topic will be covered by more items. A larger pool of 
items could be released, and some items retained for test equating 
over time. Valid jurisdiction and student results can still be 
reported. 


Recommendation 


That the Achievement Testing Program be implemented through the 
use of matrix sampling, with multiple test forms sharing a common 
set of items. 


3. Test Security 


Issue 
Release of tests after administration 


It is relatively easy to maintain test security until it is 
administered. Once an examination is administered it can no 
longer be considered secure. Release of the tests will make 
jurisdiction results more meaningful and enhance the credibility 
of results. The tests can provide models for teacher-made tests 
in the province. Once the pool of items used on tests becomes 
large, some previously used items can be used on tests without 
invalidating results. 


Recommendation 


That achievement tests not be secured after they have been 
administered. 
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4. Test Equating 


Issue 


Providing for comparison of test results between different tests 
in a particular subject area 


It is part of the mandate of SEB to monitor achievement on a 
long-term basis. This is taken to mean that trends in student 
achievement over several years should be monitored and described. 
To accomplish this, equating procedures will be required to 
compare one year's performance in a given subject area to that of 
a previous year or administration. If tests are not secured after 
administration, this cannot be accomplished by repeating items on 
successive tests. Therefore, equivalent items must be constructed 
and equated at the field testing stage. If matrix sampling is 
used, a small number of items could be secured for test equating 
purposes. Once the item pools become large, a small number of 
previously released items can be used for test equating. 


Recommendation 


That test equating be accomplished through field testing 
procedures. 
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5. Standard Setting 


Issue 
Defining "Standards" 


The notion of standards is a deceptively simple one in that when 
we talk about standards in education, we usually make the 
assumption that we all mean the same thing. When we refer to 
standards of student performance, we are talking about a "measure 
of what (student performance) is adequate for some purpose." When 
educators are required to set measurable performance standards, 
the complexity becomes apparent. When measurable standards are 
defined as the requirement, a quest for sensible standard setting 
procedures may begin. Until a common understanding of the 
function and nature of standards is achieved, it is not possible 
to agree on how they are to be measured. 


Recommendation 


That SEB clarify and communicate its position regarding standards 
of student performance. 


Issue 
Procedures for establishing acceptable levels of achievement 


Now that educators are being called on to deal openly with the 
question of standards, they realize there is no generally 
acceptable procedure for setting standards for achievement tests. 
While the SEB can supply provincial averages to which 
jurisdictions can compare their results, these averages are not 
sufficient as standards because they do not provide a basis for 
judging whether provincial performance is acceptable, nor do they 
allow for comparisons of achievement between different subtests. 
The performance of a jurisdiction is not necessarily satisfactory 
by virtue of the fact that its average is above the provincial 
average, nor unsatisfactory if its average is lower than the 
provincial average. Therefore some attempt must be made to 
establish standards that are not solely based on test results. 
This procedure should involve teachers because they are in an 
excellent position to judge what can be expected of students on 
particular examinations. 


Alberta 


EDUCATION 


Devonian Building, West Tower, 11160 Jasper Avenue, Edmonton, Alberta, Canada T5K OL2 


June 16, 1983 


TO: ALL MEMBERS OF THE AD HOC STUDENT 
EVALUATION TECHNICAL ADVISORY COMMITTEE 


RE: FINAL REPORT OF THE TECHNICAL ADVISORY COMMITTEE 


Enclosed is a copy of the final report of the Ad Hoc Student 
Evaluation Technical Advisory Committee submitted to Dr. Lloyd Symyrozum, 
Director of Student Evaluation, Alberta Education. 


I wish to take this opportunity to thank you on behalf of 
Dr. Symyrozum and myself for your enthusiastic and constructive 
contributions in preparing this report. The recommendations put forth by 
the Committee will have significant implications for testing programs of 
Alberta Education, and your invaluable contributions are greatly 
appreciated. 


I would also like to extend to you my personal thanks for 
accepting the invitation to participate in this project. I am confident 
I share the feelings of all when I say that I found the experience 
enjoyable as well as productive. 


Once again, thank you all for giving so generously of your time 
and effort. Have a good summer. 


J. Lawrence Tymko 

Chairman 

Ad Hoc Student Evaluation Technical 
Advisory Committee 
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Recommendations 


1) That SEB adopt some procedure involving teachers for setting 
standards for all achievement tests. 


That the TIFSS proposal (Appendix II) be tested ona trial 
basis. 


6. Reports 


Issue 
Reporting of results 


Release of reports composed entirely of statistical information 
leaves consumers without a basis for interpretation. Data tables 
are difficult to understand without some description, and results 
are subject to misinterpretation without guidelines. The results 
ought to be reported in a fashion that makes clear the 
significance of the results to the various publics. 


Recommendation 


That reports of results include guidelines for interpretation and 
use. 


7. Achievement Testing Schedule 


Issue 

Scheduling of achievement tests 

Administration of two achievement tests at the same grade level in 
a given year poses problems. Because of the time required, the 


motivation problems may be compounded. This may also result in 
teacher objections. 


Recommendation 


That any grade be required to write only one achievement test in 
a given year. 


8. Research in Testing 


8.1 Issue 


Providing for the investigation of operational and technical 
problems in testing 


There are many new developments in measurement with potential for 
improving the validity and utility of achievement tests. In some 
cases the new techniques can be applied directly, but in most 
cases they need to be evaluated and adapted to SEB requirements. 
In addition, there are certain operational problems that require 
investigation in order to reach satisfactory solution. 
Feasibility studies are required so that resources are applied 
appropriately. For example, the following areas need 
investigation: 


- item format and scoring procedures 

- sampling procedures 

- standard setting 

- item calibration 

- alternate approaches to establishing validity 

- effectiveness of alternative reporting procedures 
- test equating 


Recommendation 


That the SEB conduct internal projects to satisfy immediate needs 
and supervise external projects to satisfy long range needs. 
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APPENDIX I 


AD HOC STUDENT EVALUATION TECHNICAL ADVISORY COMMITTEE 


An ad hoc technical advisory committee will be established for 


the Student Evaluation Branch. 


Purpose 

The purpose of the Committee will be to review the technical 
aspects of the Student Evaluation Program and to make such 
recommendations as it deems appropriate to the Director of Student 


Evaluation of Alberta Education. 


Terms of Reference 

The Committee, under the chairmanship of an Associate Director 
of the Student Evaluation Branch, will review and advise on any technical 
aspect of student evaluation referred to it by the Director of Student 


Evaluation, Alberta Education. 


Structure of the Committee 
The Committee will be composed of: 


1. A chairman and two other members appointed by the Director of 
Student Evaluation 


2. A selected representative from each of the University of 
Alberta, the University of Calgary, and the University of 
Lethbridge 


3. A selected representative from the Alberta School Trustees' 
Association 


4. A selected representative from the Alberta Teachers' Association 


5. A selected representative from each of the Edmonton and Calgary 
public school districts 


Background 

During the past two years, the Student Evaluation Branch has 
constructed, administered, and reported on one round of the Achievement 
Testing Program, and is in to the first round of the Comprehensive 
Examinations Program. At the present time, revisions are being made to 
both programs, and therefore it appears to be an appropriate time to 
conduct a review of the technical aspects of the student evaluation 


programs. 


Purpose 

The purpose of the technical advisory committee is to examine 
the technical aspects of the student evaluation programs and to make 
recommendations for future practice. Amongst the aspects to be examined 


are the following: 


A. The process of test construction, including: 
1. the development and use of curriculum specifications 
2. the validity of test blueprints 


3. the appropriateness and usefulness of field testing, and the 
use of field test information in item revision 


4. the quality of items as evidenced by content validity 
information, difficulty levels, discrimination indices 


5. the quality of the finished instrument, including time 
limits, test statistics, test efficiency, and validity 


B. Validity and efficiency of administration and scoring, including: 
1. timing of the test in the academic year 
2. distribution of test materials 


3. sampling procedures 


4. process of returning completed examinations 
5. reliability of scoring (essay and multiple-choice) 
6. validity of scoring 
7. efficiency of scoring 
C. Utility and appropriateness of reporting procedures, including: 
1. utility of standards 
2. validity of standards and interpretation bands 
3. readability of reports 
4. timeliness of reports 


D. Psychometric quality of tests 


Procedures 

The Ad Hoc Technical Advisory Committee will be composed of 
people who are knowledgeable and interested in psychometrics. They will 
undertake to review the work of the Student Evaluation Branch from a 
psychometric perspective, keeping in mind curriculum concerns. It is 
anticipated that most, if not all, of the materials to be reviewed are 
available. In some instances it may be necessary to interview people who 
are involved in various test construction projects, but for the most 
part, members of the advisory committee who are on staff in the Branch 
will be able to provide the needed information. 

In making recommendations, the ad hoc advisory committee should 
be mindful of the policy, budget, and time constraints that surround the 
student evaluation programs and try to be as specific and practical as 


possible. 


The review process will involve general meetings to discuss the 
testing programs as a whole, and subgroup meetings to deal with the 


various tests and their results. 


Timeline 

A preliminary report should be prepared for the Director by 
May 15, 1983. Following this, discussions between the ad hoc committee 
and the Director should take place at which recommendations can be 


revised. The final report will be completed by June 15, 1983. 
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APPENDIX IT 


TIFSS: A Proposal 
(Teacher Input For Setting Standards) 


A continuing problem in our achievement testing program is the 
determination of standards against which to interpret test results. The 
people who should be most knowledgeable about the level of achievement that 
can be reasonably expected from students in a course are those teachers 
teaching the course. The purpose of this document is to outline a procedure 
which will involve a large number of teachers in the standard setting process. 

It is proposed that all teachers whose class or classes are writing an 
achievement test be required to complete a form on which they would record the 
following for a portion of the multiple-choice items on the test: 

A. The alternative the teacher believes to be correct; 

B. The percentage (0-20, 21~40, 41-60, 61-80, 81-100) of students the 

teacher believes will select the correct alternative; and 

C. The percentage (same scale as B) of students taking the course in the 

province that should be able to select the correct alternative. 

It is important to have teachers identify the correct response in order 
to ensure that they read the question and alternatives carefully. This will 
give them a better understanding of what is required of the students to answer 
the question. 

Each teacher would respond to 15 - 25 items. There will be three to five 
color-coded versions of the teacher response form for each test. Any one 
school will receive only one version of the teacher responses form and the 
versions will be randomly distributed among schools. Each school will receive 
one teacher response form per class. This procedure should ensure that at 
least 120 teachers will rate each item. 

The resulting data can be used in conjunction with test results and the 
judgments of other professionals to help determine acceptable levels of 


achievement on the test. 


ITEM RATING FORM 


Grade 6 Mathematics 


bert 


EDUCATION 


In this year's testing program, we would like to involve all practising 
teachers in the determination of standards against which results may be 
interpreted. We feel that teachers of a given course are the most 
knowledgeable about the level of achievement that can be reasonably expected 
of the students in that course. Therefore, we are asking each teacher who has 
a class writing a test to estimate expected levels of achievement for some 
multiple-choice items. You are asked to respond to the set of items 
identified on the next page. These estimates will be considered in the 
setting of target averages for each of the subtests. We are also asking 
teachers to identify the correct response as a further check on item validity. 


Instructions 
Please fill out the following information on the attached form: 


1. Place a check mark in the appropriate space to indicate whether your class 
is above average, average, or below average. If you teach more than one 
class, think of all your students as a single group and fill out the form 
for that larger group. 


2. Give the following information for each multiple-choice item identified: 


a) Circle the number below the letter corresponding to the correct 
response. 


b) Estimate the percentage of your students that you feel will choose the 
correct response. Circle the number below the appropriate percentage 
range under the heading "% in class that will get item correct." 


c) Estimate the percentage of students in the province that you feel 
should be able to choose the correct response. Circle the number 
below the appropriate percentage range under the heading "% in 
province that should get item correct." 


3. If you wish to comment on the test or on the item rating procedure, please 
use the attached comment sheet. 


Thank you for your assistance in setting standards for the interpretation of 
the achievement test results. 


Please give this form to your principal so that it can be returned with the 
achievement tests to the Student Evaluation Branch. 


61 


61 


My class is ___—s«— (11): above average ____ (2) average _____ (3) below average 
Item No. Correct Response % in class that % in province that 

will get item correct should get item correct 
O- 21- 41- 61- §81- O- 21- 41- 61- 81- 
A B Cc D 20 40 60 80 = 100 20 40 60 80 100 

1. 1 2 3 4 1 2 3 4 5 1 2 3 4 5 

2. 1 2 3 4 1 2 3 4 5 1 2 3 4 5 

3. 1 2 3 4 1 2 3 4 5 1 2 3 4 5 

4. 1 2 3 4 1 2 3 4 5 1 2 3 4 5 

5 L. @ a 1 2 3 4 °5 1 2 3 4 5 

6. 1 2 3 4 1 2 3 4 5 1 2 3 4 5 

7 1 2 3 4 1 2 3 4 5 1 2 3 4 5 

8. 1 2 3 4 1 2 3 4 5 1 2 3 4 5 

9. 1 2 3 4 1 2 3 4 5 1 2 3 4 5 

10 1 2 3 4 1 2 3 4 5 1 2 3 4 5 

11 1 2 3 4 1 2 3 4 5 1 2 3 4 5 

12 1 2 3 4 1 2 3 4 5 1 2 3 4 5 

13 1 2 3 4 1 2 3 4 5 1 2 3 4 5 

14 1 2 3 4 1 2 3 4 5 1 2 3 4 5 


COMMENTS 


APPENDIX IIT 


EXAMINATION DEVELOPMENT PROCESS 
STUDENT EVALUATION BRANCH 


ALBERTA EDUCATION 


From the time an examination is conceived to the time it is administered, 
many individuals are involved: practising classroom teachers, student 
evaluation personnel, curriculum and field services personnel, 
educational administrators, and representatives of post-secondary 
institutions. Writing questions and structuring examinations are 
extremely complex tasks requiring extended periods of time for 
completion. Following is a brief summary of the various phases of the 


test development procedure used by the Student Evaluation Branch, Alberta 


Education. 


PHASE ONE: PLANNING 


The first phase in the development of an examination begins with a review 
of the learning objectives prescribed by Alberta Education through a 

pare ianiac program of study and an assessment of the appropriate 
curriculum specifications prepared under the direction of the Curriculum 


Branch. 

Test developers then prepare tentative examination specifications 
(specific learning objectives to be tested) through an interim blueprint 
(an overall plan used to guide the development of an examination). 
Questions posed by test developers at this point are: 


- Who will take the test and for what purposes? 


- What types of questions will constitute the examination (machine 


scored, short answer, or extended written response)? 


- What knowledge, skills, and attitudes can students be expected to 


possess? 
- How long and how demanding should the examination be? 


'- How and to whom should the results of the examination be reported? 


So that examination results will be reliable and can be reported 
meaningfully, multiple-choice sections of the examinations must contain a 
minimum of six items testing any particular type, or domain, of a 
learning objective. In addition, the range of difficulty of 
mntisiesdieice items is expected to vary from .30 to .85*, and the 


overall mean of such items is expected to be .625*. 


Once decisions have been made concerning blueprints, and the choice of 
item forms and examination format has been determined, test developers 
plan procedures for item development. First, existing items are checked 
for "fit" against the interim blueprint and any deficiencies in the item 
pool are noted. Then, committees of classroom teachers practising at the 
appropriate grade level are formed, and meeting dates are arranged. Test 
developers prepare manuals and in-service procedures that will guide the 
work of these committees in the development of items that will eliminate 


any noted deficiencies in the pool of existing items. 


After agreement within the Student Evaluation Branch, the interim 
blueprint is brought before an inter-branch committee consisting of the 
appropriate Associate Director of Curriculum (or of Language Services), 
Field Services consultants, and the members of the test development 
team. This committee reviews the planning that has been carried out and 


makes recommendations. 


* Generally, no fewer than 30 per cent of the students will 
answer the most difficult item correctly, and no more than 85 per cent 
of the students will answer the easiest item correctly. 


** The expected mean of an examination or test section consisting of 
multiple-choice items containing four alternatives should be 62.5 per 
cent, the mid point between chance selection (25 per cent) and 
perfection (100 per cent). 
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PHASE TWO: BLUEPRINT APPROVAL 


Once there is agreement within Alberta Education as to the proposed 
nature of the examination, the blueprint is reviewed and recommended for 
approval by a Technical Review Committee consisting of members nominated 
by various stakeholder groups (the Alberta Teachers' Association, the 
Conference of Alberta School Superintendents, the post-secondary 
institutions, and Alberta Education). This committee makes 
recommendations to the Director of Student Evaluation. Its role is 


advisory in nature. 


At this point in the procedure, an information bulletin describing the 
examination, including sample questions and items, is prepared. This 
bulletin is meant for distribution to educators, and to the public at 


large, well in advance of the administration date of the examination. 


PHASE THREE: ITEM DEVELOPMENT 


The committees of practising classroom teachers formed during the 
planning phase of the examination development procedure now meet, under 
the supervision of Student Evaluation Branch personnel, to develop new 
items. Where necessary, item builders are trained in the principles of 
item constuction. Once they have had in-service training, item builders 
work as a group or individually and, when possible, meet to review, 


revise, and validate the items they have written. 


New items built in commitee are then screened for format, validity, 
blueprint "fit," and other design considerations. Copyright approval is 
sought for testing stimuli such as literary passages, political cartoons, 


graphs, and charts. 


PHASE FOUR: FIELD TESTING 


Items are then compiled and placed in field tests which are submitted to 
a strict regimen of quality control including further validity checking, 
editing, proofreading, and key checking. Any required art work is 


completed during this phase of the procedure. 


With appropriate permission, field testing is carried out on a sample of 
the student population for which the examination has been designed, 
usually intact classes which have completed the portion of the curriculum 
covered by the field test. A desired minimum of 150 students writes each 


field test. 


The results of each field test are then analyzed and the statistics or 
item analyses are scrutinized to determine whether or not individual 


items require further revision. 


PHASE FIVE: ITEM REVISION AND VALIDATION 


Items judged to require further changes on the basis of field test 


results are revised and resubmitted for further field testing. 


Field tested items having potential for future use are given a further 
validity check, this time by a validation committee consisting of 
classroom teachers practising at the appropriate grade level and a 


representative from a post-secondary institution. 


PHASE SIX: PILOT TEST CONSTRUCTION 


Once a pool of validated items is available, a pilot test is constructed 
which follows the approved blueprint and parallels the final examination 


in content, format, and design. 


Pilot tests, like field tests, are submitted to a strict regimen of 
quality control - validity checking, editing, proofreading, and key 


checking. 


After the pilot test has been constructed, it is brought before a second 
meeting of the inter-branch committee, consisting of the appropriate 
Associate Director of Curriculum (or of Language Services), Field 
Services consultants, and the members of the test development team. This 


committee reviews the pilot test and makes recommendations. 


PHASE SEVEN: PILOT TEST APPROVAL 


The Technical Review Committee which has approved the examination 
blueprint in Phase Two of the Examination Development Procedure meets a 
second time to review and recommend for approval the pilot test and 
administration instructions. It is at this point that the Technical 
Review Committee sets standards of achievement appropriate for the pilot 
test, including marking standards, if the test includes short answer 
items or extended writing. Again, this committee makes recommendations 


to the Director of Student Evaluation. 


PHASE EIGHT: PILOT TESTING 


The pilot test is then administered to an appropriate sample of the 
student population for which the examination has been developed, one 
which has received the proper level of instruction, is properly 
motivated, and is representative of the ability distribution in the 


population. A minimum of 250 students writes the pilot test. 


Final revisions are made on the basis of information collected as a 
result of the pilot test administration. If major revisions have been 
made, the Technical Review Committee is reconvened to scrutinize these 


changes. 

The information bulletin prepared during Phase Two of the Examination 

Development Procedure is now finalized, printed, and distributed. 
PHASE NINE: PREPARATION AND ADMINISTRATION OF THE 


FINAL EXAMINATION FORM 


The examination is submitted for final quality control procedures, 


including editing, proofreading, and a final key checking. 


The examination is then sent to a commercial printer, prepared, 


distributed for administration, and administered. 


PHASE TEN: ANALYZING AND REPORTING THE RESULTS 


The results of the final examination are then analyzed. The Technical 
Review Committee reconvenes to review. and recommend for approval, the 
results of the examination. The standards of achievement previously set 


may be reconsidered in light of the results. 


A variety of reports are prepared — the Summary Report, the Provincial 
Report, Jurisdiction Reports, and the Branch Technical File. These 


reports vary in depth of reporting and in intended audience. 


TEST DEVELOPMENT PROCEDURE TIMELINES 


SPECIFIC EXAMPLE GENERAL NEEDS 
TEST DEVELOPMENT PHASES (GRADE 9 L.A. ACHIEVEMENT) (ACHIEVEMENT TESTS) 
PHASE ONE: PLANNING NOVEMBER 1982 - 
MONTH 1 - 3 
PHASE TWO: BLUEPRINT JANUARY 1983 
APPROVAL 
PHASE THREE: ITEM FEBRUARY — MAY 1983 MONTH 4 - 7 
DEVELOPMENT 
PHASE FOUR: FIELD TESTING JUNE 1983 MONTH 8 
PHASE FIVE: ITEM REVISION JULY - SEPTEMBER 1983 MONTH 93 - 11 
AND VALIDATION 
PHASE SIX: PILOT TEST _ 
CONSTRUCTION 
OCTOBER 1983 MONTH 12 
PHASE SEVEN: PILOT TEST 
APPROVAL 
PHASE EIGHT: PILOT TESTING JUNE 1984 MONTH 20 
PHASE NINE: CONSTRUCT AND JUNE 1985 MONTH 32 
ADMINISTER FINAL 
EXAM FORM 
PHASE TEN: ANALYZING AND JULY - AUGUST 1985 MONTH 33 - 34 


REPORTING RESULTS 
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APPENDIX IV 
TECHNICAL REVIEW COMMITTEES 
FOR THE 


ACHIEVEMENT TESTING PROGRAM 


A Technical Review Committee will be established for each of the 
following major subject areas: 
Language Arts Science 


Mathematics Social Studies 


Purpose of the Committees 


The purpose of each committee will be to review the total examination 
process (i.e., development, administration, scoring, and reporting) and 
make such recommendations as they deem appropriate to the Director of 


Student Evaluation of Alberta Education. 


Terms of Reference 
Each committee, under the chairmanship of an Associate Director of the 
Student Evaluation Branch, will carry out the following responsibilities: 
1. Review and recommend for approval the following for each 
Achievement Test to be administered: 
(a) a blueprint and standards for the test 
(b) general administration of the test 
(c) scoring system for written response portion 
(d) final draft of the test 
2. Review the results and standards of the Achievement Test. 
3. Consider any other matters referred to it by the Director of the 
Student Evaluation Branch. 
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Structure of the Committees 


Each committee will be composed of: 


1. 


2 


A chairman appointed by the Director of Student Evaluation. 

A member nominated hy the Assistant Deputy Minister of Program 
Development. 

A member nominated by the Assistant Deputy Minister of Program 
Delivery. 

A selected representative of post-secondary institutions. 

Two members nominated by the Alberta Teachers' Association who 
have demonstrated an interest and expertise in the subject area, 
but who are not engaged in teaching the specific course heing 
examined. 

A member nominated by the Conference of Alberta School 


Superintendents. 


Term of Office 


Each Technical Review Committeee will be a standing committee of the 


Student Evaluation Branch. Each member will hold office for a term of 


one, two, or three years as determined hy the Director of Student 


Evaluation. At the conclusion of a member's term, the respective 


organization will be invited to review their representation and to 


reconfirm or resubmit their nomination. 


STUDENT EVALUATION BRANCH 


March 10, 1983 
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